Data-based decision rules about the convexity of the support of a distribution

Given n independent, identically distributed random vectors in R, drawn from a common density f , one wishes to find out whether the support of f is convex or not. In this paper we describe a decision rule which decides correctly for sufficiently large n, with probability 1, whenever f is bounded away from zero in its compact support. We also show that the assumption of boundedness is necessary. The rule is based on a statistic that is a second-order U -statistic with a random kernel. Moreover, we suggest a way of approximating the distribution of the statistic under the hypothesis of convexity of the support. The performance of the proposed method is illustrated on simulated data sets. As an example of its potential statistical implications, the decision rule is used to automatically choose the tuning parameter of ISOMAP, a nonlinear dimensionality reduction method.